You should document this change in https://docs.python.org/dev/whatsnew/3.14.html

Done.

FWIW, in Debian we plan to backport this to 3.13 too.

I'm only aware of Debian who uses "multiarch". Do other operating systems also use it? Maybe Debian variants, Ubuntu, and Ubuntu variants?

This change will slow down any "import module". I don't recall if there is a cache for that or not.

Maybe Debian variants, Ubuntu, and Ubuntu variants?

All Debian derivatives, yes. They typically don't deviate very much, when it comes to plumbing.

This change will slow down any "import module". I don't recall if there is a cache for that or not.

What do you want to do about that? Is there a benchmark you'd like to see results for?

I see a table of 5 entries (including NULL) increasing to 6. That is one extra item to search, when:

1. Importing C extensions from multiarch filenames.
2. Importing C extensions with no tag of any kind (foo.so).
3. Failing to import something because there isn't a C extension with this name.

What do you want to do about that? Is there a benchmark you'd like to see results for?

Example of command:

strace -o trace python3 -c pass

strace output:

newfstatat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__init__.cpython-312-x86_64-linux-gnu.so", 0x7ffe20002e50, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__init__.abi3.so", 0x7ffe20002e50, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__init__.so", 0x7ffe20002e50, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__init__.py", {st_mode=S_IFREG|0644, st_size=5884, ...}, 0) = 0

newfstatat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__init__.py", {st_mode=S_IFREG|0644, st_size=5884, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__pycache__/__init__.cpython-312.pyc", O_RDONLY|O_CLOEXEC) = 3

Internally, when Python imports the encoding package, it has to do 4 fstat() calls. You can see a check on the .abi3.so suffix.

If we add a new entry to the import suffix, it will add one fstat() syscall per import (at least, to import a package). We should consider the impact on performance if we decide to add this feature.

I'm only aware of Debian who uses "multiarch". Do other operating systems also use it? Maybe Debian variants, Ubuntu, and Ubuntu variants?

FWIW: macOS, iOS and Android all use multiarch as a configuration value - it's used to differentiate ARM from x86_64 simulators, and devices from simulators/emulators.

However, on iOS and Android, there's limited need to keep binary artefacts in the same folder, as any given executable can only have a single architecture's executables. In addition, in the case of iOS, the binaries need to be migrated to the Frameworks folder and named as Frameworks, so any "side-by-side" benefits would be lost. Any "other platform" executables need to be stripped out as part of the build process, at which point there's no naming conflict.

macOS uses the multiarch config value, but the value is always "darwin", and the universal binary format exists to support multiple architectures in a single binary file. I guess it might be useful to be able to have x86_64 and ARM64 binaries side-by-side... but there's probably only a year or two left in the official supported life of x86_64, so I don't think adding this feature will ultimately be that helpful for the macOS use case.

@stefanor: Did you consider to maintain this change as a downstream-only patch in Debian?

If you would like to make it upstream, I would suggest making it optional, disabled by default, and add a configure option to enable it. That's how I added some Fedora specific changes, such as:

• https://docs.python.org/dev/using/configure.html#cmdoption-with-platlibdir
• https://docs.python.org/dev/using/configure.html#cmdoption-with-wheel-pkg-dir

Here are some benchmarks:

Google Sheet with Data

There is no discernable performance difference in minimal interpreter startup python -c ''.

import time
import sys
from subprocess import check_call

t1 = time.perf_counter()
for i in range(1000):
    check_call([sys.executable, "-c", ""])
t2 = time.perf_counter()
print((t2-t1) / 1000)

Looking at strace, I see only a single extra syscall.

We can manufacture an import-intensive benchmark:

import pkgutil
import time

t1 = time.perf_counter()
for module in pkgutil.walk_packages(onerror=lambda pkg: None):
    if module.name.startswith("test."):
        continue
    if module.name.endswith(".__main__"):
        continue
    if module.name in {"antigravity", "idlelib.idle", "this", "zen"}:
        continue
    try:
        __import__(module)
    except Exception:
        pass
t2 = time.perf_counter()
print((t2 - t1))

Analysing this, I see 116 stat syscalls on .so filenames jumping to 232. But in a 100ms total execution time, that's negligible, and I can't see any statistically significant results.

I'm only aware of Debian who uses "multiarch".

Also, note that this is used on all linux platforms. The default for (non-stable ABI) extensions is to include the multiarch tuple in the extension filename. Pick a random binary wheel built in manylinux, and you'll see them.

@stefanor: Did you consider to maintain this change as a downstream-only patch in Debian?

I would be happy to do that. Although I prefer not carrying downstream-only patches long-term if possible. Look at the mess around dist-packages for example.

Without this MR, there's no real point in supporting the multiarch tags in the non-stable ABI extensions. Either we get the benefit from doing it everywhere, or you say you don't need to support the feature, and we rip it out everywhere. We've got half a feature at the moment. Would you prefer it to all be behind a config argument?

If you would like to make it upstream, I would suggest making it optional, disabled by default, and add a configure option to enable it. That's how I added some Fedora specific changes, such as:

These are not entirely Fedora-specific. And that's a good argument for always trying to upstream.

https://docs.python.org/dev/using/configure.html#cmdoption-with-platlibdir

In Debian we use those paths for our cross-compilers. I imagine there is a scenario where it's useful to have Python headers in there.

https://docs.python.org/dev/using/configure.html#cmdoption-with-wheel-pkg-dir

We're a happy customer of this too, It meant one less patch to carry.

There is no discernable performance difference in minimal interpreter startup python -c ''.

Maybe 0.5% slowdown with PGO. For 1 extra syscall, I'm surprised it's visible. This may still not be significant.

@erlend-aasland @encukou: Would you mind to have a look at this issue? What do you think? Should it become the default behavior, or should it be a configure option?

IMO, the most important thing here is to keep this in mind for abi4 (or whatever we name the free-threading-compatible stable ABI). There it should be the only option.
If we get abi4 in 3.14, it's probably fine to keep this as a downstream-only patch for one more release.

IMO, the most important thing here is to keep this in mind for abi4 (or whatever we name the free-threading-compatible stable ABI). There it should be the only option.

Yeah, that sounds sensible.

If we get abi4 in 3.14, it's probably fine to keep this as a downstream-only patch for one more release.

Is that expected? We've been on abi3 for quite a while.

There it should be the only option.

If we want that for the stable ABI, do we want to do the same thing for the regular ABI? It would probably affect a lot of build systems that assume they can name their output extension.so

Is that expected? We've been on abi3 for quite a while.

Yes, over a decade. abi3 showing its age, and since it's incompatible free-threading we'll need a new alternative soon.
abi3 is not going away, but changing it this late in the cycle might not be worth it.

If we want that for the stable ABI, do we want to do the same thing for the regular ABI?

I don't see much reason to deprecate and remove that, so that we don't break the build systems you mention. (Note that even abi3/abi4 extensions will work with a bare .so extension, since the stable ABI is a subset of the full one.)

FWIW, in Debian we plan to backport this to 3.13 too.

I'm going to include it in our 3.13.0 upload.

🔔 (I'd like to remind people to please consider this)

Right now, there is no concrete plan for an abi4.

We can't put the non-stable-ABI ones behind a configuration option, because that would break compatibility with existing wheels.

Can't you build Python with such new ./configure option on Debian?

We can. My point is that the world currently publishes wheels that contain multiarch tuples in extension filenames (that don't use the stable ABI). So the world's interpreters need to be able to import these.

So, if you change this behaviour, it needs to be in a new Python release.

Just did a no-change rebase to get this mergable again.

🔔 any chance we can get this merged for 3.15?

Did this get consensus on the forum?

It has not been raised on the forum. There has been discussion here and in the issue, but it seems to have stalled.

This PR is stale because it has been open for 30 days with no activity.

Documentation build overview

📚 cpython-previews | 🛠️ Build #32748615 | 📁 Comparing 3e17279 against main (a7ed0c9)

  🔍 Preview build  

But, if this is not a default, then it is not safe for us to ever name C extensions with a multiarch suffix, the way we do for non-stable-ABI ones, in public distributions. Personally, I'd prefer to be consistent about this. At least for abi4.

AFAIK, you are the only ones shipping such extensions, no?

AFAIK, you are the only ones shipping such extensions, no?

At the moment, we're the only ones with this patch. So, the only ones producing such extensions.

But, adding it and making it default (which this branch now does), means all Linux have a multiarch tag in linux abi3 and abi3t extensions, the same way they do for non-stable-ABI extensions.

But that's something IMO we want to discourage...

If you need to share platlib between installations due to the Debian guidelines, fine, but that's not something we really want people doing. Separating platlib is the solution people should be using, if at all possible, with your approach only as a last resort.

I am okay with supporting your use-case in a best-effort capacity, but I do not think it should be the default behavior.

That said, I don't want to block this PR, so if the rest of the maintainers agree, please go ahead.

The only thing I ask is to add tests.

If you need to share platlib between installations due to the Debian guidelines, fine, but that's not something we really want people doing

If that's the case, PEP3149 should be overturned. We implemented PEP3149, to allow us to do this. But stable-abi was forgotten, and when people started to use them, we ran into problems and filed this MR. From my position, PEP3149 works well, and I don't see any need to discourage its use.

I am okay with supporting your use-case in a best-effort capacity, but I do not think it should be the default behavior.

It's currently the default behaviour for non-stable ABI. Why should stable-abi be different? Surely we want to align?